Skip to content

Key EPContext cache by compile identity - #1295

Open
Qiong Wu (qiowu) (DingmaomaoBJTU) wants to merge 1 commit into
mainfrom
dingmaomaobjtu-epcontext-cache-key
Open

Key EPContext cache by compile identity#1295
Qiong Wu (qiowu) (DingmaomaoBJTU) wants to merge 1 commit into
mainfrom
dingmaomaobjtu-epcontext-cache-key

Conversation

@DingmaomaoBJTU

Copy link
Copy Markdown
Collaborator

Summary

  • replace direct WinMLSession mtime-only EPContext reuse with a strict compile identity
  • compile cache misses into immutable UUID generations selected by an atomic marker
  • serialize same-identity compilation across threads/processes while isolating different identities
  • hash source ONNX/external-data bytes, EP DLL, hardware, effective options, embed mode, ORT version, and generated artifacts
  • publish stable CLI outputs atomically with content-addressed binary references while retaining the legacy stable binary alias

Problem

WinMLSession.compile() previously reused <model>_<device>_ctx.onnx whenever it was newer than the source ONNX. The cache key omitted the EP/source/version, hardware, provider/session options, embed mode, ORT version, external-data sidecars, and context binary integrity. Different configurations could therefore reuse or overwrite one shared EPContext artifact.

Behavior

  • legacy or malformed caches without an identity marker rebuild once
  • exact identities reuse an immutable generation
  • source/external data, EP DLL, hardware, provider/session options, embed mode, or ORT changes produce a different cache namespace
  • source changes observed while waiting or compiling are retried and never published under a stale identity
  • custom opaque SessionOptions and uncertain source identity use unique non-cacheable generations
  • marker/artifact failures fail closed without discarding a successfully compiled generation
  • missing, modified, or timestamp-preserving substituted context binaries invalidate reuse
  • compiler finalization consumes the exact session generation and never scans stale siblings after raw fallback
  • public ONNX output remains stable, references an immutable content-addressed binary, and still publishes the documented stable binary alias

Validation

  • affected unit suites: 125 passed, 6 existing skips
  • focused cache/concurrency suite: 16 passed
  • Ruff check and format check on all changed Python files
  • mypy on all changed production modules
  • real QNN/NPU keen_hominy compile:
    • first compile produced one identity marker and immutable generation
    • second compile logged Using cached EPContext
    • second compile reduced total time from about 13.0s to 1.75s
    • public model_qnn_ctx.onnx references model_qnn_ctx_qnn.<sha256>.bin
    • legacy model_qnn_ctx_qnn.bin alias remains present with matching size

uv lock --offline --check could not resolve uncached Linux onnx metadata on this ARM64 host. The lockfile TOML parses and the added unit test verifies that every direct project dependency is represented in both the editable package dependencies and requires-dist metadata; CI runs on the supported AMD64 environment.

ep_device=ep_device,
ep_config=ep_config,
)
running_model_path = model_path

if TYPE_CHECKING:
from pathlib import Path
import onnx
onnx_path=simple_matmul_onnx,
ep_device=qnn_npu_ep_device,
ep_config=EPConfig(provider="qnn", enable_ep_context=True),
session_options=lambda: MagicMock(),
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants